Document scoring based on traffic associated with a document

ABSTRACT

A system determines an extent to which advertisements are presented or updated within a document, a quality of an advertiser associated with an advertisement provided within the document, whether an advertisement in the document relates to an advertising document that has more than a threshold amount of traffic, and/or an extent to which an advertisement provided within the document generates user traffic to an advertising document related to the advertisement. The system generates a score for the document based, at least in part, on the extent to which advertisements are presented or updated, the quality of the advertiser associated with the advertisement, whether the advertisement relates to an advertising document that has more than the threshold amount of traffic, and/or the extent to which the advertisement generates user traffic to the advertising document. The system ranks the document with regard to at least one other document based on the score.

RELATED APPLICATION

This application is a divisional of U.S. patent application Ser. No.10/748,664, filed Dec. 31, 2003, which claims priority under 35 U.S.C.§119 based on U.S. Provisional Application No. 60/507,617, filed Sep.30, 2003, the disclosures of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to information retrieval systemsand, more particularly, to systems and methods for generating searchresults based, at least in part, on historical data associated withrelevant documents.

2. Description of Related Art

The World Wide Web (“web”) contains a vast amount of information. Searchengines assist users in locating desired portions of this information bycataloging web documents. Typically, in response to a user's request, asearch engine returns links to documents relevant to the request.

Search engines may base their determination of the user's interest onsearch terms (called a search query) provided by the user. The goal of asearch engine is to identify links to high quality relevant resultsbased on the search query. Typically, the search engine accomplishesthis by matching the terms in the search query to a corpus of pre-storedweb documents. Web documents that contain the user's search terms areconsidered “hits” and are returned to the user.

Ideally, a search engine, in response to a given user's search query,will provide the user with the most relevant results. One category ofsearch engines identifies relevant documents based on a comparison ofthe search query terms to the words contained in the documents. Anothercategory of search engines identifies relevant documents using factorsother than, or in addition to, the presence of the search query terms inthe documents. One such search engine uses information associated withlinks to or from the documents to determine the relative importance ofthe documents.

Both categories of search engines strive to provide high quality resultsfor a search query. There are several factors that may affect thequality of the results generated by a search engine. For example, someweb site producers use spamming techniques to artificially inflate theirrank. Also, “stale” documents (i.e., those documents that have not beenupdated for a period of time and, thus, contain stale data) may beranked higher than “fresher” documents (i.e., those documents that havebeen more recently updated and, thus, contain more recent data). In someparticular contexts, the higher ranking stale documents degrade thesearch results.

Thus, there remains a need to improve the quality of results generatedby search engines.

SUMMARY OF THE INVENTION

Systems and methods consistent with the principles of the invention mayscore documents based, at least in part, on history data associated withthe documents. This scoring may be used to improve search resultsgenerated in connection with a search query.

According to one aspect, a method may include determining an extent towhich advertisements are presented or updated within a document;generating a score for the document based, at least in part, on theextent to which advertisements are presented or updated within thedocument; and ranking the document with regard to at least one otherdocument based, at least in part, on the score.

According to another aspect, a system may include means for determininga quality of an advertiser associated with an advertisement providedwithin a document; means for generating a score for the document based,at least in part, on the quality of the advertiser associated with theadvertisement provided within the document; and means for ranking thedocument with regard to at least one other document based, at least inpart, on the score.

According to yet another aspect, a method may include determiningwhether an advertisement in a document relates to an advertisingdocument that has more than a threshold amount of traffic; generating ascore for the document based, at least in part, on whether theadvertisement relates to an advertising document that has more than thethreshold amount of traffic; and ranking the document with regard to atleast one other document based, at least in part, on the score.

According to a further aspect, a method may include determining anextent to which an advertisement provided within a document generatesuser traffic to an advertising document related to the advertisement;generating a score for the document based, at least in part, on theextent to which the advertisement generates user traffic to theadvertising document; and ranking the document with regard to at leastone other document based, at least in part, on the score.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate an embodiment of the inventionand, together with the description, explain the invention. In thedrawings,

FIG. 1 is a diagram of an exemplary network in which systems and methodsconsistent with the principles of the invention may be implemented;

FIG. 2 is an exemplary diagram of a client and/or server of FIG. 1according to an implementation consistent with the principles of theinvention;

FIG. 3 is an exemplary functional block diagram of the search engine ofFIG. 1 according to an implementation consistent with the principles ofthe invention; and

FIG. 4 is a flowchart of exemplary processing for scoring documentsaccording to an implementation consistent with the principles of theinvention.

DETAILED DESCRIPTION

The following detailed description of the invention refers to theaccompanying drawings. The same reference numbers in different drawingsmay identify the same or similar elements. Also, the following detaileddescription does not limit the invention.

Systems and methods consistent with the principles of the invention mayscore documents using, for example, history data associated with thedocuments. The systems and methods may use these scores to provide highquality search results.

A “document,” as the term is used herein, is to be broadly interpretedto include any machine-readable and machine-storable work product. Adocument may include an e-mail, a web site, a file, a combination offiles, one or more files with embedded links to other files, a newsgroup posting, a blog, a web advertisement, etc. In the context of theInternet, a common document is a web page. Web pages often includetextual information and may include embedded information (such as metainformation, images, hyperlinks, etc.) and/or embedded instructions(such as Javascript, etc.). A page may correspond to a document or aportion of a document. Therefore, the words “page” and “document” may beused interchangeably in some cases. In other cases, a page may refer toa portion of a document, such as a sub-document. It may also be possiblefor a page to correspond to more than a single document.

In the description to follow, documents may be described as having linksto other documents and/or links from other documents. For example, whena document includes a link to another document, the link may be referredto as a “forward link.” When a document includes a link from anotherdocument, the link may be referred to as a “back link.” When the term“link” is used, it may refer to either a back link or a forward link.

Exemplary Network Configuration

FIG. 1 is an exemplary diagram of a network 100 in which systems andmethods consistent with the principles of the invention may beimplemented. Network 100 may include multiple clients 110 connected tomultiple servers 120-140 via a network 150. Network 150 may include alocal area network (LAN), a wide area network (WAN), a telephonenetwork, such as the Public Switched Telephone Network (PSTN), anintranet, the Internet, a memory device, another type of network, or acombination of networks. Two clients 110 and three servers 120-140 havebeen illustrated as connected to network 150 for simplicity. Inpractice, there may be more or fewer clients and servers. Also, in someinstances, a client may perform the functions of a server and a servermay perform the functions of a client.

Clients 110 may include client entities. An entity may be defined as adevice, such as a wireless telephone, a personal computer, a personaldigital assistant (PDA), a lap top, or another type of computation orcommunication device, a thread or process running on one of thesedevices, and/or an object executable by one of these device. Servers120-140 may include server entities that gather, process, search, and/ormaintain documents in a manner consistent with the principles of theinvention. Clients 110 and servers 120-140 may connect to network 150via wired, wireless, and/or optical connections.

In an implementation consistent with the principles of the invention,server 120 may include a search engine 125 usable by clients 110. Server120 may crawl a corpus of documents (e.g., web pages), index thedocuments, and store information associated with the documents in arepository of crawled documents. Servers 130 and 140 may store ormaintain documents that may be crawled by server 120. While servers120-140 are shown as separate entities, it may be possible for one ormore of servers 120-140 to perform one or more of the functions ofanother one or more of servers 120-140. For example, it may be possiblethat two or more of servers 120-140 are implemented as a single server.It may also be possible for a single one of servers 120-140 to beimplemented as two or more separate (and possibly distributed) devices.

Exemplary Client/Server Architecture

FIG. 2 is an exemplary diagram of a client or server entity (hereinaftercalled “client/server entity”), which may correspond to one or more ofclients 110 and servers 120-140, according to an implementationconsistent with the principles of the invention. The client/serverentity may include a bus 210, a processor 220, a main memory 230, a readonly memory (ROM) 240, a storage device 250, one or more input devices260, one or more output devices 270, and a communication interface 280.Bus 210 may include one or more conductors that permit communicationamong the components of the client/server entity.

Processor 220 may include one or more conventional processors ormicroprocessors that interpret and execute instructions. Main memory 230may include a random access memory (RAM) or another type of dynamicstorage device that stores information and instructions for execution byprocessor 220. ROM 240 may include a conventional ROM device or anothertype of static storage device that stores static information andinstructions for use by processor 220. Storage device 250 may include amagnetic and/or optical recording medium and its corresponding drive.

Input device(s) 260 may include one or more conventional mechanisms thatpermit an operator to input information to the client/server entity,such as a keyboard, a mouse, a pen, voice recognition and/or biometricmechanisms, etc. Output device(s) 270 may include one or moreconventional mechanisms that output information to the operator,including a display, a printer, a speaker, etc. Communication interface280 may include any transceiver-like mechanism that enables theclient/server entity to communicate with other devices and/or systems.For example, communication interface 280 may include mechanisms forcommunicating with another device or system via a network, such asnetwork 150.

As will be described in detail below, the client/server entity,consistent with the principles of the invention, perform certainsearching-related operations. The client/server entity may perform theseoperations in response to processor 220 executing software instructionscontained in a computer-readable medium, such as memory 230. Acomputer-readable medium may be defined as one or more physical orlogical memory devices and/or carrier waves.

The software instructions may be read into memory 230 from anothercomputer-readable medium, such as data storage device 250, or fromanother device via communication interface 280. The softwareinstructions contained in memory 230 may cause processor 220 to performprocesses that will be described later. Alternatively, hardwiredcircuitry may be used in place of or in combination with softwareinstructions to implement processes consistent with the principles ofthe invention. Thus, implementations consistent with the principles ofthe invention are not limited to any specific combination of hardwarecircuitry and software.

Exemplary Search Engine

FIG. 3 is an exemplary functional block diagram of search engine 125according to an implementation consistent with the principles of theinvention. Search engine 125 may include document locator 310, historycomponent 320, and ranking component 330. As shown in FIG. 3, one ormore of document locator 310 and history component 320 may connect to adocument corpus 340. Document corpus 340 may include informationassociated with documents that were previously crawled, indexed, andstored, for example, in a database accessible by search engine 125.History data, as will be described in more detail below, may beassociated with each of the documents in document corpus 340. Thehistory data may be stored in document corpus 340 or elsewhere.

Document locator 310 may identify a set of documents whose contentsmatch a user search query. Document locator 310 may initially locatedocuments from document corpus 340 by comparing the terms in the user'ssearch query to the documents in the corpus. In general, processes forindexing documents and searching the indexed collection to return a setof documents containing the searched terms are well known in the art.Accordingly, this functionality of document locator 310 will not bedescribed further herein.

History component 320 may gather history data associated with thedocuments in document corpus 340. In implementations consistent with theprinciples of the invention, the history data may include data relatingto: document inception dates; document content updates/changes; queryanalysis; link-based criteria; anchor text (e.g., the text in which ahyperlink is embedded, typically underlined or otherwise highlighted ina document); traffic; user behavior; domain-related information; rankinghistory; user maintained/generated data (e.g., bookmarks); unique words,bigrams, and phrases in anchor text; linkage of independent peers;and/or document topics. These different types of history data aredescribed in additional detail below. In other implementations, thehistory data may include additional or different kinds of data.

Ranking component 330 may assign a ranking score (also called simply a“score” herein) to one or more documents in document corpus 340. Rankingcomponent 330 may assign the ranking scores prior to, independent of, orin connection with a search query. When the documents are associatedwith a search query (e.g., identified as relevant to the search query),search engine 125 may sort the documents based on the ranking score andreturn the sorted set of documents to the client that submitted thesearch query. Consistent with aspects of the invention, the rankingscore is a value that attempts to quantify the quality of the documents.In implementations consistent with the principles of the invention, thescore is based, at least in part, on the history data from historycomponent 320.

Exemplary History Data Document Inception Date

According to an implementation consistent with the principles of theinvention, a document's inception date may be used to generate (oralter) a score associated with that document. The term “date” is usedbroadly here and may, thus, include time and date measurements. Asdescribed below, there are several techniques that can be used todetermine a document's inception date. Some of these techniques are“biased” in the sense that they can be influenced by third partiesdesiring to improve the score associated with a document. Othertechniques are not biased. Any of these techniques, combinations ofthese techniques, or yet other techniques may be used to determine adocument's inception date.

According to one implementation, the inception date of a document may bedetermined from the date that search engine 125 first learns of orindexes the document. Search engine 125 may discover the documentthrough crawling, submission of the document (or arepresentation/summary thereof) to search engine 125 from an “outside”source, a combination of crawl or submission-based indexing techniques,or in other ways. Alternatively, the inception date of a document may bedetermined from the date that search engine 125 first discovers a linkto the document.

According to another implementation, the date that a domain with which adocument is registered may be used as an indication of the inceptiondate of the document. According to yet another implementation, the firsttime that a document is referenced in another document, such as a newsarticle, newsgroup, mailing list, or a combination of one or more suchdocuments, may be used to infer an inception date of the document.According to a further implementation, the date that a document includesat least a threshold number of pages may be used as an indication of theinception date of the document. According to another implementation, theinception date of a document may be equal to a time stamp associatedwith the document by the server hosting the document. Other techniques,not specifically mentioned herein, or combinations of techniques couldbe used to determine or infer a document's inception date.

Search engine 125 may use the inception date of a document for scoringof the document. For example, it may be assumed that a document with afairly recent inception date will not have a significant number of linksfrom other documents (i.e., back links). For existing link-based scoringtechniques that score based on the number of links to/from a document,this recent document may be scored lower than an older document that hasa larger number of links (e.g., back links). When the inception date ofthe documents are considered, however, the scores of the documents maybe modified (either positively or negatively) based on the documents'inception dates.

Consider the example of a document with an inception date of yesterdaythat is referenced by 10 back links. This document may be scored higherby search engine 125 than a document with an inception date of 10 yearsago that is referenced by 100 back links because the rate of link growthfor the former is relatively higher than the latter. While a spiky rateof growth in the number of back links may be a factor used by searchengine 125 to score documents, it may also signal an attempt to spamsearch engine 125. Accordingly, in this situation, search engine 125 mayactually lower the score of a document(s) to reduce the effect ofspamming.

Thus, according to an implementation consistent with the principles ofthe invention, search engine 125 may use the inception date of adocument to determine a rate at which links to the document are created(e.g., as an average per unit time based on the number of links createdsince the inception date or some window in that period). This rate canthen be used to score the document, for example, giving more weight todocuments to which links are generated more often.

In one implementation, search engine 125 may modify the link-based scoreof a document as follows:

H=L/log(F+2),

where H may refer to the history-adjusted link score, L may refer to thelink score given to the document, which can be derived using any knownlink scoring technique (e.g., the scoring technique described in U.S.Pat. No. 6,285,999) that assigns a score to a document based on linksto/from the document, and F may refer to elapsed time measured from theinception date associated with the document (or a window within thisperiod).

For some queries, older documents may be more favorable than newer ones.As a result, it may be beneficial to adjust the score of a documentbased on the difference (in age) from the average age of the result set.In other words, search engine 125 may determine the age of each of thedocuments in a result set (e.g., using their inception dates), determinethe average age of the documents, and modify the scores of the documents(either positively or negatively) based on a difference between thedocuments' age and the average age.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information relating to theinception date of the document.

Content Updates/Changes

According to an implementation consistent with the principles of theinvention, information relating to a manner in which a document'scontent changes over time may be used to generate (or alter) a scoreassociated with that document. For example, a document whose content isedited often may be scored differently than a document whose contentremains static over time. Also, a document having a relatively largeamount of its content updated over time might be scored differently thana document having a relatively small amount of its content updated overtime.

In one implementation, search engine 125 may generate a content updatescore (U) as follows:

U=f(UF,UA),

where f may refer to a function, such as a sum or weighted sum, UF mayrefer to an update frequency score that represents how often a document(or page) is updated, and UA may refer to an update amount score thatrepresents how much the document (or page) has changed over time. UF maybe determined in a number of ways, including as an average time betweenupdates, the number of updates in a given time period, etc.

UA may also be determined as a function of one or more factors, such asthe number of “new” or unique pages associated with a document over aperiod of time. Another factor might include the ratio of the number ofnew or unique pages associated with a document over a period of timeversus the total number of pages associated with that document. Yetanother factor may include the amount that the document is updated overone or more periods of time (e.g., n % of a document's visible contentmay change over a period t (e.g., last m months)), which might be anaverage value. A further factor might include the amount that thedocument (or page) has changed in one or more periods of time (e.g.,within the last x days).

According to one exemplary implementation, UA may be determined as afunction of differently weighted portions of document content. Forinstance, content deemed to be unimportant if updated/changed, such asJavascript, comments, advertisements, navigational elements, boilerplatematerial, or date/time tags, may be given relatively little weight oreven ignored altogether when determining UA. On the other hand, contentdeemed to be important if updated/changed (e.g., more often, morerecently, more extensively, etc.), such as the title or anchor textassociated with the forward links, could be given more weight thanchanges to other content when determining UA.

UF and UA may be used in other ways to influence the score assigned to adocument. For example, the rate of change in a current time period canbe compared to the rate of change in another (e.g., previous) timeperiod to determine whether there is an acceleration or decelerationtrend. Documents for which there is an increase in the rate of changemight be scored higher than those documents for which there is a steadyrate of change, even if that rate of change is relatively high. Theamount of change may also be a factor in this scoring. For example,documents for which there is an increase in the rate of change when thatamount of change is greater than some threshold might be scored higherthan those documents for which there is a steady rate of change or anamount of change is less than the threshold.

In some situations, data storage resources may be insufficient to storethe documents when monitoring the documents for content changes. In thiscase, search engine 125 may store representations of the documents andmonitor these representations for changes. For example, search engine125 may store “signatures” of documents instead of the (entire)documents themselves to detect changes to document content. In thiscase, search engine 125 may store a term vector for a document (or page)and monitor it for relatively large changes. According to anotherimplementation, search engine 125 may store and monitor a relativelysmall portion (e.g., a few terms) of the documents that are determinedto be important or the most frequently occurring (excluding “stopwords”).

According to yet another implementation, search engine 125 may store asummary or other representation of a document and monitor thisinformation for changes. According to a further implementation, searchengine 125 may generate a similarity hash (which may be used to detectnear-duplication of a document) for the document and monitor it forchanges. A change in a similarity hash may be considered to indicate arelatively large change in its associated document. In otherimplementations, yet other techniques may be used to monitor documentsfor changes. In situations where adequate data storage resources exist,the full documents may be stored and used to determine changes ratherthan some representation of the documents.

For some queries, documents with content that has not recently changedmay be more favorable than documents with content that has recentlychanged. As a result, it may be beneficial to adjust the score of adocument based on the difference from the average date-of-change of theresult set. In other words, search engine 125 may determine a date whenthe content of each of the documents in a result set last changed,determine the average date of change for the documents, and modify thescores of the documents (either positively or negatively) based on adifference between the documents' date-of-change and the averagedate-of-change.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information relating to amanner in which the document's content changes over time. For very largedocuments that include content belonging to multiple individuals ororganizations, the score may correspond to each of the sub-documents(i.e., that content belonging to or updated by a single individual ororganization).

Query Analysis

According to an implementation consistent with the principles of theinvention, one or more query-based factors may be used to generate (oralter) a score associated with a document. For example, one query-basedfactor may relate to the extent to which a document is selected overtime when the document is included in a set of search results. In thiscase, search engine 125 might score documents selected relatively moreoften/increasingly by users higher than other documents.

Another query-based factor may relate to the occurrence of certainsearch terms appearing in queries over time. A particular set of searchterms may increasingly appear in queries over a period of time. Forexample, terms relating to a “hot” topic that is gaining/has gainedpopularity or a breaking news event would conceivably appear frequentlyover a period of time. In this case, search engine 125 may scoredocuments associated with these search terms (or queries) higher thandocuments not associated with these terms.

A further query-based factor may relate to a change over time in thenumber of search results generated by similar queries. A significantincrease in the number of search results generated by similar queries,for example, might indicate a hot topic or breaking news and causesearch engine 125 to increase the scores of documents related to suchqueries.

Another query-based factor may relate to queries that remain relativelyconstant over time but lead to results that change over time. Forexample, a query relating to “world series champion” leads to searchresults that change over time (e.g., documents relating to a particularteam dominate search results in a given year or time of year). Thischange can be monitored and used to score documents accordingly.

Yet another query-based factor might relate to the “staleness” ofdocuments returned as search results. The staleness of a document may bebased on factors, such as document creation date, anchor growth,traffic, content change, forward/back link growth, etc. For somequeries, recent documents are very important (e.g., if searching forFrequently Asked Questions (FAQ) files, the most recent version would behighly desirable). Search engine 125 may learn which queries recentchanges are most important for by analyzing which documents in searchresults are selected by users. More specifically, search engine 125 mayconsider how often users favor a more recent document that is rankedlower than an older document in the search results. Additionally, ifover time a particular document is included in mostly topical queries(e.g., “World Series Champions”) versus more specific queries (e.g.,“New York Yankees”), then this query-based factor—by itself or withothers mentioned herein—may be used to lower a score for a document thatappears to be stale.

In some situations, a stale document may be considered more favorablethan more recent documents. As a result, search engine 125 may considerthe extent to which a document is selected over time when generating ascore for the document. For example, if for a given query, users overtime tend to select a lower ranked, relatively stale, document over ahigher ranked, relatively recent document, this may be used by searchengine 125 as an indication to adjust a score of the stale document.

Yet another query-based factor may relate to the extent to which adocument appears in results for different queries. In other words, theentropy of queries for one or more documents may be monitored and usedas a basis for scoring. For example, if a particular document appears asa hit for a discordant set of queries, this may (though not necessarily)be considered a signal that the document is spam, in which case searchengine 125 may score the document relatively lower.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on one or more query-basedfactors.

Link-Based Criteria

According to an implementation consistent with the principles of theinvention, one or more link-based factors may be used to generate (oralter) a score associated with a document. In one implementation, thelink-based factors may relate to the dates that new links appear to adocument and that existing links disappear. The appearance date of alink may be the first date that search engine 125 finds the link or thedate of the document that contains the link (e.g., the date that thedocument was found with the link or the date that it was last updated).The disappearance date of a link may be the first date that the documentcontaining the link either dropped the link or disappeared itself.

These dates may be determined by search engine 125 during a crawl orindex update operation. Using this date as a reference, search engine125 may then monitor the time-varying behavior of links to the document,such as when links appear or disappear, the rate at which links appearor disappear over time, how many links appear or disappear during agiven time period, whether there is trend toward appearance of new linksversus disappearance of existing links to the document, etc.

Using the time-varying behavior of links to (and/or from) a document,search engine 125 may score the document accordingly. For example, adownward trend in the number or rate of new links (e.g., based on acomparison of the number or rate of new links in a recent time periodversus an older time period) over time could signal to search engine 125that a document is stale, in which case search engine 125 may decreasethe document's score. Conversely, an upward trend may signal a “fresh”document (e.g., a document whose content is fresh—recently created orupdated) that might be considered more relevant, depending on theparticular situation and implementation.

By analyzing the change in the number or rate of increase/decrease ofback links to a document (or page) over time, search engine 125 mayderive a valuable signal of how fresh the document is. For example, ifsuch analysis is reflected by a curve that is dropping off, this maysignal that the document may be stale (e.g., no longer updated,diminished in importance, superceded by another document, etc.).

According to one implementation, the analysis may depend on the numberof new links to a document. For example, search engine 125 may monitorthe number of new links to a document in the last n days compared to thenumber of new links since the document was first found. Alternatively,search engine 125 may determine the oldest age of the most recent y % oflinks compared to the age of the first link found.

For the purpose of illustration, consider y=10 and two documents (websites in this example) that were both first found 100 days ago. For thefirst site, 10% of the links were found less than 10 days ago, while forthe second site 0% of the links were found less than 10 days ago (inother words, they were all found earlier). In this case, the metricresults in 0.1 for site A and 0 for site B. The metric may be scaledappropriately. In another exemplary implementation, the metric may bemodified by performing a relatively more detailed analysis of thedistribution of link dates. For example, models may be built thatpredict if a particular distribution signifies a particular type of site(e.g., a site that is no longer updated, increasing or decreasing inpopularity, superceded, etc.).

According to another implementation, the analysis may depend on weightsassigned to the links. In this case, each link may be weighted by afunction that increases with the freshness of the link. The freshness ofa link may be determined by the date of appearance/change of the link,the date of appearance/change of anchor text associated with the link,date of appearance/change of the document containing the link. The dateof appearance/change of the document containing a link may be a betterindicator of the freshness of the link based on the theory that a goodlink may go unchanged when a document gets updated if it is stillrelevant and good. In order to not update every link's freshness from aminor edit of a tiny unrelated part of a document, each updated documentmay be tested for significant changes (e.g., changes to a large portionof the document or changes to many different portions of the document)and a link's freshness may be updated (or not updated) accordingly.

Links may be weighted in other ways. For example, links may be weightedbased on how much the documents containing the links are trusted (e.g.,government documents can be given high trust). Links may also, oralternatively, be weighted based on how authoritative the documentscontaining the links are (e.g., authoritative documents may bedetermined in a manner similar to that described in U.S. Pat. No.6,285,999). Links may also, or alternatively, be weighted based on thefreshness of the documents containing the links using some otherfeatures to establish freshness (e.g., a document that is updatedfrequently (e.g., the Yahoo home page) suddenly drops a link to adocument).

Search engine 125 may raise or lower the score of a document to whichthere are links as a function of the sum of the weights of the linkspointing to it. This technique may be employed recursively. For example,assume that a document S is 2 years olds. Document S may be consideredfresh if n % of the links to S are fresh or if the documents containingforward links to S are considered fresh. The latter can be checked byusing the creation date of the document and applying this techniquerecursively.

According to yet another technique, the analysis may depend on an agedistribution associated with the links pointing to a document. In otherwords, the dates that the links to a document were created may bedetermined and input to a function that determines the age distribution.It may be assumed that the age distribution of a stale document will bevery different from the age distribution of a fresh document. Searchengine 125 may then score documents based, at least in part, on the agedistributions associated with the documents.

The dates that links appear can also be used to detect “spam,” whereowners of documents or their colleagues create links to their owndocument for the purpose of boosting the score assigned by a searchengine. A typical, “legitimate” document attracts back links slowly. Alarge spike in the quantity of back links may signal a topicalphenomenon (e.g., the CDC web site may develop many links quickly afteran outbreak, such as SARS), or signal attempts to spam a search engine(to obtain a higher ranking and, thus, better placement in searchresults) by exchanging links, purchasing links, or gaining links fromdocuments without editorial discretion on making links. Examples ofdocuments that give links without editorial discretion include guestbooks, referrer logs, and “free for all” pages that let anyone add alink to a document.

According to a further implementation, the analysis may depend on thedate that links disappear. The disappearance of many links can mean thatthe document to which these links point is stale (e.g., no longer beingupdated or has been superseded by another document). For example, searchengine 125 may monitor the date at which one or more links to a documentdisappear, the number of links that disappear in a given window of time,or some other time-varying decrease in the number of links (orlinks/updates to the documents containing such links) to a document toidentify documents that may be considered stale. Once a document hasbeen determined to be stale, the links contained in that document may bediscounted or ignored by search engine 125 when determining scores fordocuments pointed to by the links.

According to another implementation, the analysis may depend, not onlyon the age of the links to a document, but also on the dynamic-ness ofthe links. As such, search engine 125 may weight documents that have adifferent featured link each day, despite having a very fresh link,differently (e.g., lower) than documents that are consistently updatedand consistently link to a given target document. In one exemplaryimplementation, search engine 125 may generate a score for a documentbased on the scores of the documents with links to the document for allversions of the documents within a window of time. Another version ofthis may factor a discount/decay into the integration based on the majorupdate times of the document.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on one or more link-basedfactors.

Anchor Text

According to an implementation consistent with the principles of theinvention, information relating to a manner in which anchor text changesover time may be used to generate (or alter) a score associated with adocument. For example, changes over time in anchor text associated withlinks to a document may be used as an indication that there has been anupdate or even a change of focus in the document.

Alternatively, if the content of a document changes such that it differssignificantly from the anchor text associated with its back links, thenthe domain associated with the document may have changed significantly(completely) from a previous incarnation. This may occur when a domainexpires and a different party purchases the domain. Because anchor textis often considered to be part of the document to which its associatedlink points, the domain may show up in search results for queries thatare no longer on topic. This is an undesirable result.

One way to address this problem is to estimate the date that a domainchanged its focus. This may be done by determining a date when the textof a document changes significantly or when the text of the anchor textchanges significantly. All links and/or anchor text prior to that datemay then be ignored or discounted.

The freshness of anchor text may also be used as a factor in scoringdocuments. The freshness of an anchor text may be determined, forexample, by the date of appearance/change of the anchor text, the dateof appearance/change of the link associated with the anchor text, and/orthe date of appearance/change of the document to which the associatedlink points. The date of appearance/change of the document pointed to bythe link may be a good indicator of the freshness of the anchor textbased on the theory that good anchor text may go unchanged when adocument gets updated if it is still relevant and good. In order to notupdate an anchor text's freshness from a minor edit of a tiny unrelatedpart of a document, each updated document may be tested for significantchanges (e.g., changes to a large portion of the document or changes tomany different portions of the document) and an anchor text's freshnessmay be updated (or not updated) accordingly.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information relating to amanner in which anchor text changes over time.

Traffic

According to an implementation consistent with the principles of theinvention, information relating to traffic associated with a documentover time may be used to generate (or alter) a score associated with thedocument. For example, search engine 125 may monitor the time-varyingcharacteristics of traffic to, or other “use” of, a document by one ormore users. A large reduction in traffic may indicate that a documentmay be stale (e.g., no longer be updated or may be superseded by anotherdocument).

In one implementation, search engine 125 may compare the average trafficfor a document over the last j days (e.g., where j=30) to the averagetraffic during the month where the document received the most traffic,optionally adjusted for seasonal changes, or during the last k days(e.g., where k=365). Optionally, search engine 125 may identifyrepeating traffic patterns or perhaps a change in traffic patterns overtime. It may be discovered that there are periods when a document ismore or less popular (i.e., has more or less traffic), such as duringthe summer months, on weekends, or during some other seasonal timeperiod. By identifying repeating traffic patterns or changes in trafficpatterns, search engine 125 may appropriately adjust its scoring of thedocument during and outside of these periods.

Additionally, or alternatively, search engine 125 may monitortime-varying characteristics relating to “advertising traffic” for aparticular document. For example, search engine 125 may monitor one or acombination of the following factors: (1) the extent to and rate atwhich advertisements are presented or updated by a given document overtime; (2) the quality of the advertisers (e.g., a document whoseadvertisements refer/link to documents known to search engine 125 overtime to have relatively high traffic and trust, such as amazon.com, maybe given relatively more weight than those documents whoseadvertisements refer to low traffic/untrustworthy documents, such as apornographic site); and (3) the extent to which the advertisementsgenerate user traffic to the documents to which they relate (e.g., theirclick-through rate). Search engine 125 may use these time-varyingcharacteristics relating to advertising traffic to score the document.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information relating totraffic associated with the document over time.

User Behavior

According to an implementation consistent with the principles of theinvention, information corresponding to individual or aggregate userbehavior relating to a document over time may be used to generate (oralter) a score associated with the document. For example, search engine125 may monitor the number of times that a document is selected from aset of search results and/or the amount of time one or more users spendaccessing the document. Search engine 125 may then score the documentbased, at least in part, on this information.

If a document is returned for a certain query and over time, or within agiven time window, users spend either more or less time on average onthe document given the same or similar query, then this may be used asan indication that the document is fresh or stale, respectively. Forexample, assume that the query “Riverview swimming schedule” returns adocument with the title “Riverview Swimming Schedule.” Assume furtherthat users used to spend 30 seconds accessing it, but now every userthat selects the document only spends a few seconds accessing it. Searchengine 125 may use this information to determine that the document isstale (i.e., contains an outdated swimming schedule) and score thedocument accordingly.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information corresponding toindividual or aggregate user behavior relating to the document overtime.

Domain-Related Information

According to an implementation consistent with the principles of theinvention, information relating to a domain associated with a documentmay be used to generate (or alter) a score associated with the document.For example, search engine 125 may monitor information relating to how adocument is hosted within a computer network (e.g., the Internet, anintranet or other network or database of documents) and use thisinformation to score the document.

Individuals who attempt to deceive (spam) search engines often usethrowaway or “doorway” domains and attempt to obtain as much traffic aspossible before being caught. Information regarding the legitimacy ofthe domains may be used by search engine 125 when scoring the documentsassociated with these domains.

Certain signals may be used to distinguish between illegitimate andlegitimate domains. For example, domains can be renewed up to a periodof 10 years. Valuable (legitimate) domains are often paid for severalyears in advance, while doorway (illegitimate) domains rarely are usedfor more than a year. Therefore, the date when a domain expires in thefuture can be used as a factor in predicting the legitimacy of a domainand, thus, the documents associated therewith.

Also, or alternatively, the domain name server (DNS) record for a domainmay be monitored to predict whether a domain is legitimate. The DNSrecord contains details of who registered the domain, administrative andtechnical addresses, and the addresses of name servers (i.e., serversthat resolve the domain name into an IP address). By analyzing this dataover time for a domain, illegitimate domains may be identified. Forinstance, search engine 125 may monitor whether physically correctaddress information exists over a period of time, whether contactinformation for the domain changes relatively often, whether there is arelatively high number of changes between different name servers andhosting companies, etc. In one implementation, a list of known-badcontact information, name servers, and/or IP addresses may beidentified, stored, and used in predicting the legitimacy of a domainand, thus, the documents associated therewith.

Also, or alternatively, the age, or other information, regarding a nameserver associated with a domain may be used to predict the legitimacy ofthe domain. A “good” name server may have a mix of different domainsfrom different registrars and have a history of hosting those domains,while a “bad” name server might host mainly pornography or doorwaydomains, domains with commercial words (a common indicator of spam), orprimarily bulk domains from a single registrar, or might be brand new.The newness of a name server might not automatically be a negativefactor in determining the legitimacy of the associated domain, but incombination with other factors, such as ones described herein, it couldbe.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information relating to alegitimacy of a domain associated with the document.

Ranking History

According to an implementation consistent with the principles of theinvention, information relating to prior rankings of a document may beused to generate (or alter) a score associated with the document. Forexample, search engine 125 may monitor the time-varying ranking of adocument in response to search queries provided to search engine 125.Search engine 125 may determine that a document that jumps in rankingsacross many queries might be a topical document or it could signal anattempt to spam search engine 125.

Thus, the quantity or rate that a document moves in rankings over aperiod of time might be used to influence future scores assigned to thatdocument. In one implementation, for each set of search results, adocument may be weighted according to its position in the top N searchresults. For N=30, one example function might be [((N+1)−SLOT)/N]⁴. Inthis case, a top result may receive a score of 1.0, down to a score near0 for the Nth result.

A query set (e.g., of commercial queries) can be repeated, and documentsthat gained more than M % in the rankings may be flagged or thepercentage growth in ranking may be used as a signal in determiningscores for the documents. For example, search engine 125 may determinethat a query is likely commercial if the average (median) score of thetop results is relatively high and there is a significant amount ofchange in the top results from month to month. Search engine 125 mayalso monitor churn as an indication of a commercial query. Forcommercial queries, the likelihood of spam is higher, so search engine125 may treat documents associated therewith accordingly.

In addition to history of positions (or rankings) of documents for agiven query, search engine 125 may monitor (on a page, host, document,and/or domain basis) one or more other factors, such as the number ofqueries for which, and the rate at which (increasing/decreasing), adocument is selected as a search result over time; seasonality,burstiness, and other patterns over time that a document is selected asa search result; and/or changes in scores over time for a URL-querypair.

In addition, or alternatively, search engine 125 may monitor a number ofdocument (e.g., URL) independent query-based criteria over time. Forexample, search engine 125 may monitor the average score among a top setof results generated in response to a given query or set of queries andadjust the score of that set of results and/or other results generatedin response to the given query or set of queries. Moreover, searchengine 125 may monitor the number of results generated for a particularquery or set of queries over time. If search engine 125 determines thatthe number of results increases or that there is a change in the rate ofincrease (e.g., such an increase may be an indication of a “hot topic”or other phenomenon), search engine 125 may score those results higherin the future.

In addition, or alternatively, search engine 125 may monitor the ranksof documents over time to detect sudden spikes in the ranks of thedocuments. A spike may indicate either a topical phenomenon (e.g., a hottopic) or an attempt to spam search engine 125 by, for example, tradingor purchasing links. Search engine 125 may take measures to prevent spamattempts by, for example, employing hysteresis to allow a rank to growat a certain rate. In another implementation, the rank for a givendocument may be allowed a certain maximum threshold of growth over apredefined window of time. As a further measure to differentiate adocument related to a topical phenomenon from a spam document, searchengine 125 may consider mentions of the document in news articles,discussion groups, etc. on the theory that spam documents will not bementioned, for example, in the news. Any or a combination of thesetechniques may be used to curtail spamming attempts.

It may be possible for search engine 125 to make exceptions fordocuments that are determined to be authoritative in some respect, suchas government documents, web directories (e.g., Yahoo), and documentsthat have shown a relatively steady and high rank over time. Forexample, if an unusual spike in the number or rate of increase of linksto an authoritative document occurs, then search engine 125 may considersuch a document not to be spam and, thus, allow a relatively high oreven no threshold for (growth of) its rank (over time).

In addition, or alternatively, search engine 125 may considersignificant drops in ranks of documents as an indication that thesedocuments are “out of favor” or outdated. For example, if the rank of adocument over time drops significantly, then search engine 125 mayconsider the document as outdated and score the document accordingly.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information relating toprior rankings of the document.

User Maintained/Generated Data

According to an implementation consistent with the principles of theinvention, user maintained or generated data may be used to generate (oralter) a score associated with a document. For example, search engine125 may monitor data maintained or generated by a user, such as“bookmarks,” “favorites,” or other types of data that may provide someindication of documents favored by, or of interest to, the user. Searchengine 125 may obtain this data either directly (e.g., via a browserassistant) or indirectly (e.g., via a browser). Search engine 125 maythen analyze over time a number of bookmarks/favorites to which adocument is associated to determine the importance of the document.

Search engine 125 may also analyze upward and downward trends to add orremove the document (or more specifically, a path to the document) fromthe bookmarks/favorites lists, the rate at which the document is addedto or removed from the bookmarks/favorites lists, and/or whether thedocument is added to, deleted from, or accessed through thebookmarks/favorites lists. If a number of users are adding a particulardocument to their bookmarks/favorites lists or often accessing thedocument through such lists over time, this may be considered anindication that the document is relatively important. On the other hand,if a number of users are decreasingly accessing a document indicated intheir bookmarks/favorites list or are increasingly deleting/replacingthe path to such document from their lists, this may be taken as anindication that the document is outdated, unpopular, etc. Search engine125 may then score the documents accordingly.

In an alternative implementation, other types of user data that mayindicate an increase or decrease in user interest in a particulardocument over time may be used by search engine 125 to score thedocument. For example, the “temp” or cache files associated with userscould be monitored by search engine 125 to identify whether there is anincrease or decrease in a document being added over time. Similarly,cookies associated with a particular document might be monitored bysearch engine 125 to determine whether there is an upward or downwardtrend in interest in the document.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on user maintained or generateddata.

Unique Words, Bigrams, Phrases in Anchor Text

According to an implementation consistent with the principles of theinvention, information regarding unique words, bigrams, and phrases inanchor text may be used to generate (or alter) a score associated with adocument. For example, search engine 125 may monitor web (or link)graphs and their behavior over time and use this information forscoring, spam detection, or other purposes. Naturally developed webgraphs typically involve independent decisions. Synthetically generatedweb graphs, which are usually indicative of an intent to spam, are basedon coordinated decisions, causing the profile of growth in anchorwords/bigrams/phrases to likely be relatively spiky.

One reason for such spikiness may be the addition of a large number ofidentical anchors from many documents. Another possibility may be theaddition of deliberately different anchors from a lot of documents.Search engine 125 may monitor the anchors and factor them into scoring adocument to which their associated links point. For example, searchengine 125 may cap the impact of suspect anchors on the score of theassociated document. Alternatively, search engine 125 may use acontinuous scale for the likelihood of synthetic generation and derive amultiplicative factor to scale the score for the document.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information regarding uniquewords, bigrams, and phrases in anchor text associated with one or morelinks pointing to the document.

Linkage of Independent Peers

According to an implementation consistent with the principles of theinvention, information regarding linkage of independent peers (e.g.,unrelated documents) may be used to generate (or alter) a scoreassociated with a document.

A sudden growth in the number of apparently independent peers, incomingand/or outgoing, with a large number of links to individual documentsmay indicate a potentially synthetic web graph, which is an indicator ofan attempt to spam. This indication may be strengthened if the growthcorresponds to anchor text that is unusually coherent or discordant.This information can be used to demote the impact of such links, whenused with a link-based scoring technique, either as a binary decisionitem (e.g., demote the score by a fixed amount) or a multiplicativefactor.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on information regardinglinkage of independent peers.

Document Topics

According to an implementation consistent with the principles of theinvention, information regarding document topics may be used to generate(or alter) a score associated with a document. For example, searchengine 125 may perform topic extraction (e.g., through categorization,URL analysis, content analysis, clustering, summarization, a set ofunique low frequency words, or some other type of topic extraction).Search engine 125 may then monitor the topic(s) of a document over timeand use this information for scoring purposes.

A significant change over time in the set of topics associated with adocument may indicate that the document has changed owners and previousdocument indicators, such as score, anchor text, etc., are no longerreliable. Similarly, a spike in the number of topics could indicatespam. For example, if a particular document is associated with a set ofone or more topics over what may be considered a “stable” period of timeand then a (sudden) spike occurs in the number of topics associated withthe document, this may be an indication that the document has been takenover as a “doorway” document. Another indication may include thedisappearance of the original topics associated with the document. Ifone or more of these situations are detected, then search engine 125 mayreduce the relative score of such documents and/or the links, anchortext, or other data associated the document.

In summary, search engine 125 may generate (or alter) a score associatedwith a document based, at least in part, on changes in one or moretopics associated with the document.

Exemplary Processing

FIG. 4 is a flowchart of exemplary processing for scoring documentsaccording to an implementation consistent with the principles of theinvention. Processing may begin with server 120 identifying documents(act 410). The documents may include, for example, one or more documentsassociated with a search query, such as documents identified as relevantto the search query. Alternatively, the documents may include one ormore documents in a corpus or repository of documents that areindependent of any search query (e.g., documents that are identified bycrawling a network and stored in a repository).

Search engine 125 may obtain history data associated with the identifieddocuments (act 420). As described above, the history data may takedifferent forms. For example, the history data may include data relatingto document inception dates; document content updates/changes; queryanalysis; link-based criteria; anchor text; traffic; user behavior;domain-related information; ranking history; user maintained/generateddata (e.g., bookmarks and/or favorites); unique words, bigrams, andphrases in anchor text; linkage of independent peers; and/or documenttopics. Search engine 125 may obtain one, or a combination, of thesekinds of history data.

Search engine 125 may then score the identified documents based, atleast in part, on the history data (act 430). When the identifieddocuments are associated with a search query, search engine 125 may alsogenerate relevancy scores for the documents based, for example, on howrelevant they are to the search query. Search engine 125 may thencombine the history scores with the relevancy scores to obtain overallscores for the documents. Instead of combining the scores, search engine125 may alter the relevancy scores for the documents based on thehistory data, thereby raising or lowering the scores or, in some cases,leaving the scores the same. Alternatively, search engine 125 may scorethe documents based on the history data without generating relevancyscores. In any event, search engine 125 may score the documents usingone, or a combination, of the types of history data.

When the identified documents are associated with a search query, searchengine 125 may also form search results from the scored documents. Forexample, search engine 125 may sort the documents based on their scores.Search engine 125 may then form references to the documents, where areference might include a title of the document (which may contain ahypertext link that will direct the user, when selected, to the actualdocument) and a snippet (i.e., a text excerpt) from the document. Inother implementations, the references are formed differently. Searchengine 125 may present references corresponding to a number of thetop-scoring documents (e.g., a predetermined number of the documents,documents with scores above a threshold, all documents, etc.) to a userwho submitted the search query.

CONCLUSION

Systems and methods consistent with the principles of the invention mayuse history data to score documents and form high quality searchresults.

The foregoing description of preferred embodiments of the presentinvention provides illustration and description, but is not intended tobe exhaustive or to limit the invention to the precise form disclosed.Modifications and variations are possible in light of the aboveteachings or may be acquired from practice of the invention. Forexample, while a series of acts has been described with regard to FIG.4, the order of the acts may be modified in other implementationsconsistent with the principles of the invention. Also, non-dependentacts may be performed in parallel.

Further, it has generally been described that server 120 performs most,if not all, of the acts described with regard to the processing of FIG.4. In another implementation consistent with the principles of theinvention, one or more, or all, of the acts may be performed by anotherentity, such as another server 130 and/or 140 or client 110.

It will also be apparent to one of ordinary skill in the art thataspects of the invention, as described above, may be implemented in manydifferent forms of software, firmware, and hardware in theimplementations illustrated in the figures. The actual software code orspecialized control hardware used to implement aspects consistent withthe principles of the invention is not limiting of the presentinvention. Thus, the operation and behavior of the aspects weredescribed without reference to the specific software code—it beingunderstood that one of ordinary skill in the art would be able to designsoftware and control hardware to implement the aspects based on thedescription herein.

1. A method, comprising: determining an extent to which advertisementsare presented or updated within a document; generating a score for thedocument based, at least in part, on the extent to which advertisementsare presented or updated within the document; and ranking the documentwith regard to at least one other document based, at least in part, onthe score.
 2. The method of claim 1, wherein determining an extent towhich advertisements are presented or updated within a documentincludes: determining a rate at which the advertisements are presentedor updated within the document.
 3. The method of claim 1, furthercomprising: determining a quality of advertisers associated with theadvertisements presented or updated within the document; and whereingenerating a score for the document includes: scoring the documentbased, at least in part, on the extent to which the advertisements arepresented or updated within the document and the quality of theadvertisers associated with the advertisements.
 4. The method of claim3, wherein determining a quality of advertisers includes: determiningthe quality of the advertisers based, at least in part, on an amount oftraffic or trustworthiness of advertising documents related to theassociated advertisements.
 5. The method of claim 1, further comprising:determining whether the advertisements relate to advertising documentsthat have more than a threshold amount of traffic; and whereingenerating a score for the document includes: scoring the documentbased, at least in part, on the extent to which the advertisements arepresented or updated within the document and whether the advertisementsrelate to advertising documents that have more than the threshold amountof traffic.
 6. The method of claim 1, further comprising: determining anextent to which the advertisements generate user traffic to advertisingdocuments related to the advertisements; and wherein generating a scorefor the document includes: scoring the document based, at least in part,on the extent to which advertisements are presented or updated withinthe document and the extent to which the advertisements generate usertraffic to the advertising documents.
 7. A system, comprising: means fordetermining a quality of an advertiser associated with an advertisementprovided within a document; means for generating a score for thedocument based, at least in part, on the quality of the advertiserassociated with the advertisement provided within the document; andmeans for ranking the document with regard to at least one otherdocument based, at least in part, on the score.
 8. The system of claim7, wherein the means for determining a quality of an advertiserincludes: means for determining the quality of the advertiser based, atleast in part, on an amount of traffic or trustworthiness of anadvertising document related to the associated advertisement.
 9. Thesystem of claim 7, further comprising: means for determining an extentto which advertisements are presented or updated within the document;and wherein the means for generating a score for the document includes:means for scoring the document based, at least in part, on the qualityof the advertiser associated with the advertisement and the extent towhich advertisements are presented or updated within the document. 10.The system of claim 9, wherein the means for determining an extent towhich advertisements are presented or updated within the documentincludes: means for determining a rate at which the advertisements arepresented or updated within the document.
 11. The system of claim 7,further comprising: means for determining an extent to which theadvertisement generates user traffic to an advertising document relatedto the advertisement; and wherein the means for generating a score forthe document includes: means for scoring the document based, at least inpart, on the quality of the advertiser associated with the advertisementprovided within the document and the extent to which the advertisementgenerates user traffic to the advertising document.
 12. A method,comprising: determining whether an advertisement in a document relatesto an advertising document that has more than a threshold amount oftraffic; generating a score for the document based, at least in part, onwhether the advertisement relates to an advertising document that hasmore than the threshold amount of traffic; and ranking the document withregard to at least one other document based, at least in part, on thescore.
 13. The method of claim 12, further comprising: determining anextent to which advertisements are presented or updated within thedocument; and wherein generating a score for the document includes:scoring the document based, at least in part, on whether theadvertisement relates to an advertising document that has more than thethreshold amount of traffic and the extent to which the advertisementsare presented or updated within the document.
 14. The method of claim13, wherein determining an extent to which advertisements are presentedor updated within the document includes: determining a rate at which theadvertisements are presented or updated within the document.
 15. Themethod of claim 12, further comprising: determining an extent to whichthe advertisement generates user traffic to the advertising document;and wherein generating a score for the document includes: scoring thedocument based, at least in part, on whether the advertisement relatesto an advertising document that has more than the threshold amount oftraffic and the extent to which the advertisement generates user trafficto the advertising document. 16-20. (canceled)